Broad area geospatial object detection using autogenerated deep learning models

ABSTRACT

A system for automated geospatial image analysis comprising a deep learning model that receives orthorectified geospatial images, pre-labeled to demarcate objects of interest. The module presents marked geospatial images and a second set of unmarked, optimized, training geospatial images to a convolutional neural network. This process may be repeated so that an image analysis software module can detect multiple object types or categories. The image analysis software module receives orthorectified geospatial images from one or more geospatial image caches. Using a multi-scale sliding window submodule, image analysis software scans geospatial images, detects objects present and geospatially locates them.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/452,076 titled “BROAD AREA GEOSPATIAL OBJECT DETECTION USINGAUTOGENERATED DEEP LEARNING MODELS”, filed on Mar. 7, 2017, which is acontinuation of U.S. patent application Ser. No. 14/835,736 titled“BROAD AREA GEOSPATIAL OBJECT DETECTION USING AUTOGENERATED DEEPLEARNING MODELS”, filed on Aug. 26, 2015, and now issued as U.S. Pat.No. 9,589,210 on Mar. 3, 2017, the entire specification of each of whichis incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is in the field of image analysis, and moreparticularly in the field of the use of deep learning model computervision systems for automated object identification from geospatialimagery.

Discussion of the State of the Art

Image analysis has been an important field of technology at least sincethe period of World War 2, when extensive use of image analysis,photogrammetry, and related technologies were used in conjunction withaerial photography for intelligence and bombing damage assessmentpurposes (among others). However, the extent of the use of imageanalysis (particularly image analysis of remotely-sensed images),particularly for identifying or locating targets of interest, has alwaysbeen limited by the need for highly-trained, specialized image analystsor interpreters. The need for specialized (and expensive) skills haslimited the use of image analysis to a correspondingly limited range ofapplications (notably military, homeland defense, and law enforcement).

The market for image analysis has also historically been limited by thehigh cost of obtaining images to analyze. In the military arena, thebenefits were sufficiently apparent that large numbers of militaryreconnaissance flights have been made over regions of interest sinceWorld War 2. But the cost of such flights virtually excluded allcommercial applications of image analysis. Starting in the 1970s withthe Landsat satellite, this began to change as low resolution satelliteimages became publicly available. A series of new satellites has openedup progressively more applications as the resolution, spectral coverage,geographic coverage, and cost per image have all continuously improved;accordingly, a significant market in commercial remote sensing imageryhas emerged. But even this market has been limited from achieving itsfull potential because of the still-present requirement for expensive,scarce image analysis talent.

One common type of geospatial image analysis task is the “search andlocate” task. In this task, one or more targets of interest need to beidentified and precisely located. A well known example of “search andlocate” is the discovery and pinpointing of warships, tanks, or othermilitary targets of interest. Recently, focused geospatial imageanalysis of geographically specific data has been used for search andrescue efforts of downed planes or lost shipping. However, these effortshave required the work of image analysts, limiting what could be done.Development of a method to identify targets of interest rapidly, usingless resources would allow the pursuit of less urgent but promisingapplications which include assessing the scope of a refugee crisis byfor example counting tents in an area of interest, analyzing the changein infrastructure in developing nations, assessing numbers of endangeredspecies, finding military hardware in areas previously not expected tocontain such equipment, identifying previously unknown airstrips orcamps where crime or terrorism may be in operation. The ability toextend “search and locate” like tasks to large geological areas andefficiently perform them repetitively over time would allow the use ofgeospatial imagery to map remote regions, to track deforestation andre-forestation and to detect natural disasters in remote areas of theworld.

The notion of computer vision, specifically the reliable identificationby a computer of particular objects has been an active pursuit withinthe field of computer science since the late 1960s. Unfortunately, untilrecently, this pursuit has met with little success except when both theobject of interest and the background against which it is presented havebeen tightly controlled. Barriers to advancement in computer objectidentification have been both technological and logical. Thetechnological barriers have been present because, like its biologicalcounterpart, computer visual processing requires computational power andamounts of memory storage that have been prohibitive up until the last15 years. Advancement in the ability to pack more transistors into thesame volume while also reducing cost and the development of suchspecialized components as the graphics processing unit, which isoptimized to perform calculations encountered during manipulation ofvisual data has brought current hardware to the point where rapid, evenreal time, object identification is possible. There has also been asignificant maturation process in how computer scientists in the fieldprogram computers to analyze the objects of interest. Some of theseearly methods have been to break each object of interest into a uniquegrouping of simple geometric shapes or to take advantage of uniqueshading patterns of each object to identify new instances of the desiredobject. All of these early attempts gave results that were extremelysensitive to such variables as lighting, exact object placement in thefield of sample, and exact object orientation, sometimes to the degreethat the object of interest was not identifiable in the original imagewithout great care. Currently, after great advancement in computercapabilities, advances in our understanding of biological vision, andadvances in computer vision theory, a method of training computers toreliably identify specific objects of interest has emerged. This methodcombines a convolutional neural network with deep learning to train thesystem to recognize an object of interest both when presented againstmany backgrounds and when the object is in different orientations. Theconvolutional neural network which consists of several layers of filterswith partial, local field interconnections between layers interspersedwith data complexity reduction pooling layers affords computer learningof object recognition with a minimum of pre-supposition on the part ofthe programmer as the convolutional neural network determines the bestfilters to use to identify the target object. Deep learning consists ofa period of “supervised learning” which uses a moderate sized set ofimages where each image contains an example of the object to identify,for example, the human face, which is clearly demarcated or “labeled”followed by a period of “unsupervised learning” on a very large numberof unlabeled images, a portion of which do not have the object toidentify present. This convolutional neural network-deep learning modelmethod has given rise to computer systems that have been reliably usedin human facial recognition, optical character recognition, andidentification of complex sets of parts during manufacturing. Indeed,the convolutional neural network-deep learning model method has beenfound so widely useful for object identification that there are multipleprogramming libraries now publicly available for download and use forthat purpose. These include, for example, the Caffe library(BerkeleyVision and Learning Center), the Torch7 library (Nagadomi) andthe cuda-covnet2 library (Alex Krizhevsky) While the convolutionalneural network-deep learning model method has been widely and verysuccessfully applied to ground based photography and video, it has notfound application in the field of geospatial image analysis.

What is needed in the art is an automated system that both identifiesand determines the precise location of a number of objects of interestfrom geospatial imagery.

SUMMARY OF THE INVENTION

The inventor has developed a system for computer the analysis ofsatellite images to geolocate one or more targets of interest, or toidentify objects or their types.

According to a preferred embodiment of the invention, a system for broadarea geospatial object detection using auto-generated deep learningmodels, comprising a deep learning model training software module and animage analysis software module. The deep learning model software module:receives training data comprising a plurality of orthorectifiedgeospatial images with a plurality of objects present therein, at leasta first subset of the plurality of objects being labeled and a secondsubset of objects being unlabeled; aggregates the training data andclassifies the training data into a plurality of predefined categories;applies one or more image modification steps to the training data, drawnfrom a set comprising artifact removal, color standardization, andresolution data determination; optimizes the training data for deeplearning model training; discards images unsuitable for trainingpurposes; and generates an object classification model from the trainingdata using a deep learning method comprising separate processing of thefirst and second subsets of the training data through a convolutionalneural network system. The image analysis software module: receivesunanalyzed, orthorectified geospatial imagery; applies one or more imagemodification steps to the unanalyzed orthorectified geospatial imagery,drawn from a set comprising artifact removal, color standardization, andresolution data determination; optimizes the orthorectified geospatialimagery for object classification; discards images unsuitable foranalysis; uses the object classification model generated above toautomatically identify and label all objects of interest in thereceived, unanalyzed orthorectified geospatial imagery, regardless ofthe orientation of that feature item within the section and accountingfor differences in item scale by using a multi-scale sliding windowalgorithm; and outputs the locations of the identified objects ofinterest in a form dictated by the parameters of the original searchrequest.

According to another preferred embodiment of the invention, a method fora system for broad area geospatial object detection using auto-generateddeep learning models, the method comprising the steps of: a) receiving aplurality of orthorectified geospatial imagery, divided into discretesegments where a plurality of objects of interest are present therein,at least a first subset of the plurality of objects of interest beinglabeled and a second subset of objects of interest being unlabeled, to adeep learning model training software module stored in a memory of andoperating on a processor of a computing device; (b) placing pre-labeledgeospatial imagery segments into pre-defined categories based upon theobjects of interest they contain; (c) applying one or more imagemodification steps drawn from a set comprising artifact removal, colorstandardization, and resolution data determination; (d) optimizing theimages for use in the deep learning model; e) discarding image segmentsdeemed unsuitable for desired purpose; (f) generating an objectclassification model from the training data using a deep learning methodcomprising separate processing of the first and second subsets of thetraining data through a convolutional neural network system operating aspart of an image analysis software module stored in a memory of andoperating on a processor of a computing device; (g) using the imageanalysis software module trained as in steps (a) through (f) to identifyfeature items of interest within unanalyzed geospatial imagery data,regardless of the feature items' orientation and accounting fordifferences in item scale by using a multi-scale sliding windowalgorithm; and (h) reporting the identities and locations of all featureitems of interest in a form dictated by the parameters of the originalsearch request.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. One skilled inthe art will recognize that the particular embodiments illustrated inthe drawings are merely exemplary, and are not intended to limit thescope of the present invention.

FIG. 1 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

FIG. 2 is a block diagram illustrating an exemplary logical architecturefor a client device, according to various embodiments of the invention.

FIG. 3 is a block diagram illustrating an exemplary architecturalarrangement of clients, servers, and external services, according tovarious embodiments of the invention.

FIG. 4 is a block diagram illustrating an exemplary overview of acomputer system as may be used in any of the various locationsthroughout the system

FIG. 5 is a diagram of an exemplary architecture for a system forautomated image analysis that uses the deep learning model and aconvolutional neural network, according to a preferred embodiment of theinvention.

FIG. 6 is a process flow diagram of a method for geospatial imageanalysis that uses the deep learning model and a convolutional neuralnetwork using a system of the invention.

FIG. 7 is an example of image obstructing cloud detection by the imagecorrection and optimization module, which is part of an automated systemfor geospatial image analysis of the invention.

FIG. 8 is a diagram that illustrates the use of a sliding window moduleto translate the coordinate system used within the cache of geospatialimages being analyzed and the geographical longitude and latitude systemas part of the automated system for geospatial image analysis of theinvention.

FIG. 9 is made up of two panels which show examples of trained imageanalysis software module identifying two types of objects of interest aspart of the automated system for geospatial image analysis of theinvention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, various systems andmethods for advanced broad area geospatial object detection usingautogenerated deep learning models.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beunderstood that these are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the inventions may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. In general,embodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the inventions, and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the particular inventions. Accordingly,those skilled in the art will recognize that one or more of theinventions may be practiced with various modifications and alterations.Particular features of one or more of the inventions may be describedwith reference to one or more particular embodiments or figures thatform a part of the present disclosure, and in which are shown, by way ofillustration, specific embodiments of one or more of the inventions. Itshould be understood, however, that such features are not limited tousage in the one or more particular embodiments or figures withreference to which they are described. The present disclosure is neithera literal description of all embodiments of one or more of theinventions nor a listing of features of one or more of the inventionsthat must be present in all embodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred. Also, steps are generally described once perembodiment, but this does not mean they must occur once, or that theymay only occur once each time a process, method, or algorithm is carriedout or executed. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described, it will be readily apparent that a single deviceor article may be used in place of the more than one device or article.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple instantiations of a mechanism unless noted otherwise.Process descriptions or blocks in figures should be understood asrepresenting modules, segments, or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process. Alternate implementations are included withinthe scope of embodiments of the present invention in which, for example,functions may be executed out of order from that shown or discussed,including substantially concurrently or in reverse order, depending onthe functionality involved, as would be understood by those havingordinary skill in the art.

Definitions

As used herein, “orthorectified geospatial image” refers to satelliteimagery of the earth that has been digitally corrected to remove terraindistortions introduced into the image by either angle of incidence of aparticular point from the center of the satellite imaging sensor orsignificant topological changes inherent to the region of the earth thatthe image depicts. This correction is accomplished using a digitalelevation model. One example of a digital elevation model in use todayis Shuttle Radar Topography Mission (SRTM) 90m DEM data set, but othersof equal or greater precision have also been created using highdefinition stereoscopic satellite imagery of the same regions as arebeing analyzed or using topographical maps of sufficient detailavailable for that region. Geospatial images used as part of theinvention may be orthorectified using digital elevation model datasetsobtained by any method known to the art.

A “database” or “data storage subsystem” (these terms may be consideredsubstantially synonymous), as used herein, is a system adapted for thelong-term storage, indexing, and retrieval of data, the retrievaltypically being via some sort of querying interface or language.“Database” may be used to refer to relational database managementsystems known in the art, but should not be considered to be limited tosuch systems. Many alternative database or data storage systemtechnologies have been, and indeed are being, introduced in the art,including but not limited to distributed non-relational data storagesystems such as Hadoop, column-oriented databases, in-memory databases,and the like. While various embodiments may preferentially employ one oranother of the various data storage subsystems available in the art (oravailable in the future), the invention should not be construed to be solimited, as any data storage architecture may be used according to theembodiments. Similarly, while in some cases one or more particular datastorage needs are described as being satisfied by separate components(for example, an expanded private capital markets database and aconfiguration database), these descriptions refer to functional uses ofdata storage systems and do not refer to their physical architecture.For instance, any group of data storage systems of databases referred toherein may be included together in a single database management systemoperating on a single machine, or they may be included in a singledatabase management system operating on a cluster of machines as isknown in the art. Similarly, any single database (such as an expandedprivate capital markets database) may be implemented on a singlemachine, on a set of machines using clustering technology, on severalmachines connected by one or more messaging systems known in the art, orin a master/slave arrangement common in the art. These examples shouldmake clear that no particular architectural approaches to databasemanagement is preferred according to the invention, and choice of datastorage technology is at the discretion of each implementer, withoutdeparting from the scope of the invention as claimed.

As used herein, “search and locate” refers to a general class of taskswherein a set of images is searched for particular classes of targets(such as buildings, tanks, railroad terminals, downed airplanes, etc.).It is common that the set of images may be searched to find more thanone class of targets (for example, to find all targets of militaryinterest), although single target class searches may also be performed(“find all cars”). The second part of the search and locate task is toprecisely locate any resulting targets of interest (where is the airbase or refugee camp?).

As used herein, “image analysis” refers to the analysis of imagesobtained from one or more image sensors; generally, a single analysistask focuses on a set of images of a single region of interest on theearth, but image analysis may be done on multiple contiguous regions ascapture by several image sensors. Satellite and aerial imagery arecommon examples of imagery that are subjected to large scale imageanalysis.

As used herein, “cache of pre-labeled geospatial images” refers to anysource of a plurality of orthorectified geospatial image segments thathave been pre-analyzed and have had instances of one or more objects ofinterest tagged or labeled in such a way that the recipient computersystem is able to associate a specific region of that image with theobject of interest for the purpose of subsequent identification of likeobjects. These images may be stored in an image database, eitherrelational or flat file, or within a directory of image files, any ofwhich may be stored on the same computer on which the images are beingused, a storage device or storage system directly connected to thatcomputer or may be on a computer or storage system connected to therecipient computer through any of the networking methods as are known inthe art.

As used herein, “cache of multi-scale geospatial images” refers to anysource of a plurality of overlapping orthorectified geospatial imagesegments that, due to optical differences at the time of capture orprocessing differences at the time of transmission, storage or analysis,show the same geographical region at different functional resolutions.There is the further requirement that the correspondence of coordinatesystem used to catalog these segments within the cache, whetherproprietary or open, to standard geographic latitude and longitudecoordinates be known so that the location being analyzed on a givenimage segment from the cache is known at all times. These images may bestored in an image database, either relational or flat file, or within adirectory of image files, any of which may be stored on the samecomputer on which the images are being used, a storage device or storagesystem directly connected to that computer or may be on a computer orstorage system connected to the recipient computer through any of thenetworking methods as are known in the art.

As used herein “image correction and optimization module” refers to aset of programming functions that during its operation receives aplurality of orthorectified geospatial images from a cache ofpre-labeled geospatial images, normalizes these images to account forimage quality differences which include but are not limited tovariations in color balance, brightness, and contrast. This module alsoanalyzes images for aberrations which might include cloud cover, lensartifact, mechanical obstruction of portions of the image and thesoftware within the module may then reject the image from analysis whencertain pre-set thresholds are exceeded.

As used herein “category” refers to a set of specific objects that areof the same type and function, but which may vary to some degree inappearance. An example of this might be the United States CapitolBuilding, the White House and the Pentagon in Washington D.C. all appeardifferent in geospatial images but are all in the category “buildings.”Another example might be that the Airbus 310, Lockheed L1011, Boeing727, Boeing 777 and Boeing 747 all differ in size and fine levelconfiguration, but are all in the category “airliners.”

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be disclosedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, and the like), aconsumer electronic device, a music player, or any other suitableelectronic device, router, switch, or the like, or any combinationthereof. In at least some embodiments, at least some of the features orfunctionalities of the various embodiments disclosed herein may beimplemented in one or more virtualized computing environments (e.g.,network computing clouds, virtual machines hosted on one or morephysical computing machines, or the like).

Referring now to FIG. 1, there is shown a block diagram depicting anexemplary computing device 100 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 100 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 100 may be adaptedto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 100 includes one or more centralprocessing units (CPU) 102, one or more interfaces 110, and one or morebusses 106 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 102may be responsible for implementing specific functions associated withthe functions of a specifically configured computing device or machine.For example, in at least one embodiment, a computing device 100 may beconfigured or designed to function as a server system utilizing CPU 102,local memory 101 and/or remote memory 120, and interface(s) 110. In atleast one embodiment, CPU 102 may be caused to perform one or more ofthe different types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 102 may include one or more processors 103 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 103 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 100. In a specificembodiment, a local memory 101 (such as non-volatile random accessmemory (RAM) and/or read-only memory (ROM), including for example one ormore levels of cached memory) may also form part of CPU 102. However,there are many different ways in which memory may be coupled to system100. Memory 101 may be used for a variety of purposes such as, forexample, caching and/or storing data, programming instructions, and thelike.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 110 are provided as network interfacecards (NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 110 may forexample support other peripherals used with computing device 100. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, Firewire, PCI, parallel, radio frequency (RF),Bluetooth, near-field communications (e.g., using near-field magnetics),802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces,Gigabit Ethernet interfaces, asynchronous transfer mode (ATM)interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale(POS) interfaces, fiber data distributed interfaces (FDDIs), and thelike. Generally, such interfaces 110 may include ports appropriate forcommunication with appropriate media. In some cases, they may alsoinclude an independent processor and, in some instances, volatile and/ornon-volatile memory (e.g., RAM).

Although the system shown in FIG. 1 illustrates one specificarchitecture for a computing device 100 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 103 may be used, and such processors 103may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 103 handlescommunications as well as routing computations, while in otherembodiments a separate dedicated communications processor may beprovided. In various embodiments, different types of features orfunctionalities may be implemented in a system according to theinvention that includes a client device (such as a tablet device orsmartphone running client software) and server systems (such as a serversystem described in more detail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 120 and local memory 101) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 120 or memories101, 120 may also be configured to store data structures, configurationdata, encryption data, historical system operations information, or anyother specific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory, solid state drives, memristormemory, random access memory (RAM), and the like. Examples of programinstructions include both object code, such as may be produced by acompiler, machine code, such as may be produced by an assembler or alinker, byte code, such as may be generated by for example a Javacompiler and may be executed using a Java virtual machine or equivalent,or files containing higher level code that may be executed by thecomputer using an interpreter (for example, scripts written in Python,Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 2,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 200 includes processors210 that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 230. Processors 210 may carry out computinginstructions under control of an operating system 220 such as, forexample, a version of Microsoft's Windows operating system, Apple's MacOS/X or iOS operating systems, some variety of the Linux operatingsystem, Google's Android operating system, or the like. In many cases,one or more shared services 225 may be operable in system 200, and maybe useful for providing common services to client applications 230.Services 225 may for example be Windows services, user-space commonservices in a Linux environment, or any other type of common servicearchitecture used with operating system 210. Input devices 270 may be ofany type suitable for receiving user input, including for example akeyboard, touchscreen, microphone (for example, for voice input), mouse,touchpad, trackball, or any combination thereof. Output devices 260 maybe of any type suitable for providing output to one or more users,whether remote or local to system 200, and may include for example oneor more screens for visual output, speakers, printers, or anycombination thereof. Memory 240 may be random-access memory having anystructure and architecture known in the art, for use by processors 210,for example to run software. Storage devices 250 may be any magnetic,optical, mechanical, memristor, or electrical storage device for storageof data in digital form. Examples of storage devices 250 include flashmemory, magnetic hard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 3, there is shown a blockdiagram depicting an exemplary architecture for implementing at least aportion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 330 may be provided. Each client 330 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 200 such as that illustrated in FIG. 2. In addition,any number of servers 320 may be provided for handling requests receivedfrom one or more clients 330. Clients 330 and servers 320 maycommunicate with one another via one or more electronic networks 310,which may be in various embodiments of the Internet, a wide areanetwork, a mobile telephony network, a wireless network (such as WiFi,Wimax, and so forth), or a local area network (or indeed any networktopology known in the art; the invention does not prefer any one networktopology over any other). Networks 310 may be implemented using anyknown network protocols, including for example wired and/or wirelessprotocols.

In addition, in some embodiments, servers 320 may call external services370 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 370 may take place, for example, via one or morenetworks 310. In various embodiments, external services 370 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 230 are implemented on a smartphone or other electronicdevice, client applications 230 may obtain information stored in aserver system 320 in the cloud or on an external service 370 deployed onone or more of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 330 or servers 320 (orboth) may make use of one or more specialized services or appliancesthat may be deployed locally or remotely across one or more networks310. For example, one or more databases 340 may be used or referred toby one or more embodiments of the invention. It should be understood byone having ordinary skill in the art that databases 340 may be arrangedin a wide variety of architectures and using a wide variety of dataaccess and manipulation means. For example, in various embodiments oneor more databases 340 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop, MapReduce, BigTable, and so forth). Insome embodiments variant database architectures such as column-orienteddatabases, in-memory databases, clustered databases, distributeddatabases, key-value stores, or even flat file data repositories may beused according to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 360 and configuration systems 350. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 360 orconfiguration 350 system or approach is specifically required by thedescription of any specific embodiment.

FIG. 4 shows an exemplary overview of a computer system 400 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 400 withoutdeparting from the broader scope of the system and method disclosedherein. CPU 401 is connected to bus 402, to which bus is also connectedmemory 403, nonvolatile memory 404, display 407, I/O unit 408, andnetwork interface card (NIC) 413. I/O unit 408 may, typically, beconnected to keyboard 409, pointing device 410, hard disk 412, andreal-time clock 411. NIC 413 connects to network 414, which may be theInternet or a local network, which local network may or may not haveconnections to the Internet. Also shown as part of system 400 is powersupply unit 405 connected, in this example, to ac supply 406. Not shownare batteries that could be present, and many other devices andmodifications that are well known but are not applicable to the specificnovel functions of the current system and method disclosed herein. Itshould be appreciated that some or all components illustrated may becombined, such as in various integrated applications (for example,Qualcomm or Samsung SOC-based devices), or whenever it may beappropriate to combine multiple capabilities or functions into a singlehardware device (for instance, in mobile devices such as smartphones,video game consoles, in-vehicle computer systems such as navigation ormultimedia systems in automobiles, or other integrated hardwaredevices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

Conceptual Architecture

FIG. 5 is a block diagram of an exemplary architecture for a system 500for automated image analysis that uses the deep learning model and aconvolutional neural network, according to a preferred embodiment of theinvention. According to the embodiment, a cache of pre-labeled,orthorectified geospatial image segments 510 is employed by deeplearning training module 520 for the purpose of training the system 500to identify feature items of interest to those running the analysis.This cache of pre-labeled geospatial image segments 510 could be storedwithin a directory or part of a database of geospatial image segmentsstored on a disk system within the computer running the deep learningmodel training module 520, stored in a directory, or in a database on adisk system directly connected to the computer running the deep learningmodel training module 520 by an external bus such as the universalserial bus or eSATA. Image cache might also be stored on one or morecomputers other than that running the deep learning model module andconnected to the deep learning module by a local internal network, theinternet or by any other means known to the art as the system does notrely on any one method of image delivery whether server-client or peerto peer in nature. The system is also not restricted to specificmechanisms by which feature items of interest found in orthorectifiedgeospatial image segments to be used in training of the system areidentified and labeled prior to use in training. One preferredembodiment might use cache 510 where feature items of interest arelabeled prior to training by many participants in a crowdsourcingcampaign to tag one or more feature items in each image segment foundwithin the cache. Image segments tagged in this way where taggingaccuracy of items is known to be high could then be added to thetraining cache for that feature item category (building, airliner,storage tank, truck). Another preferred embodiment might use cache ofgeospatial images 510 where feature items of interest are labeled bytrained image analysts, either specifically for use in the deep learningmodel training module or as part of other analyses and then re-purposedlater for deep learning model training module 520. Caches of pre-labeledgeospatial images used to train the system 500 could contain mixtures ofimage segments obtained crowdsourcing, image analyst generated effort,as well as image segments where specific feature items are identifiedand pre-labeled by any other method known to those skilled in the art,as the invention does not rely on any specific mode of identificationand labeling for deep level training model module operation.

Prior to use in the training of the image analysis software module 540image segments from the cache of pre-labeled geospatial image segments510 undergo one or more steps of digital image correction andoptimization 521. A digital correction that might be done to imagesegments to be used by deep learning model training module is conversionfrom color to grayscale as this correction reduces image segmentcomplexity, which aids in the training process. In a preferredembodiment of the system the conversion is done by first converting theimage from the RGB colorspace to the YCrCb colorspace and thendiscarding all but Y channel data which results in a grayscale image oftonal quality known to work well in deep learning model training ofconvolutional neural net used in the invention. The method of color tograyscale image conversion outlined here is meant only to be exemplaryand should not be seen to limit conversion method that could be used,including the absence of this color to grayscale conversion in thetraining image preparation process. Another type of image correctionthat may be employed to prepare pre-labeled geospatial images for use intraining is histogram normalization which often increases image contrastand serves to reduce the effects of exposure by producing sets of imagesegments with similar dynamic range profiles prior to use in training ofthe image analysis module 540. Examples of histogram normalizationfilters that may be used to prepare geospatial images for training arelinear histogram normalization, histogram equalization, adaptivehistogram equalization, and contrast limiting adaptive histogramequalization. One skilled in the art will realize that while the use ofthese image histogram manipulation methods may produce image segmentssignificantly better suited for the supervised stage of deep learning520, the system described herein does not absolutely rely on histogramnormalization in its generalized form or and of the exemplary histogrammanipulation methods specifically. In addition to or in lieu of to thosementioned in some depth, image filters such as Gaussian, median filter,and bilateral filter to enhance edge contrast may be applied topre-labeled geospatial image segments as is common in the art, howeverthe listing of these filters is meant only to provide examples andshould not be taken to bar the use of other filters that are not listedas part of the invention.

As is appreciated by those skilled in the art, much of the recentsuccess in the recognition of specific objects in digital images bycomputers using trained convolutional neural networks is due to the useof the deep learning method. Under this method of training, theconvolutional neural network is first trained with a set of images wherethe item to be identified, ideally in conjunction with a large pluralityof backgrounds and in a large plurality of orientations, is clearlydemarcated digitally. This stage of deep learning is referred to by theart as “supervised.” The number of images where the object of interesthas been clearly demarcated is often limited and often gives rise totrained convolutional neural networks that are excellent in recognizingthe object of interest in the training set but otherwise recognize theobject of interest poorly. Object recognition reliability issignificantly increased by exposure of the convolutional neural networkto sets of images, much larger in image number than the labeled-trainingset, where the object of interest is present but not labeled or wherethe object may not be present. This stage of learning is referred to bythe art as “unsupervised.” One method to generate large sets of imagesfor unsupervised learning is to screen very large repositories ofgeneral images for suitable examples. Another method of generating theunsupervised learning image set is to start with the set of images usedfor supervised learning stage and modify them by linearly translating orrotationally transforming the object of interest; changing theappearance of the object of interest or background through imagehistogram modification, Gaussian filter, bilateral filter or median blurfilter; or both using object placement and filter modification to createa plurality of images useful in the unsupervised learning stage fromeach image used in the supervised learning stage. In one preferredembodiment of the system 500 it is this second method, manipulating thegeospatial images used in the supervised learning stage to augment thetraining image set, which is depicted 522. The choice to depict this onemethod was for figure simplification only as system training by deeplearning method training module can use geospatial images generated byeither method, identification of new images, or manipulation of thesupervised learning image set, even some mixture of both methods, totrain image analysis software module 540.

The trained image analysis software module 540 can and often does havethe ability to identify multiple categories of objects of interest fromgeospatial imagery. For example a first image analysis software modulemay identify items from a set comprising buildings, tents, cars,airliners, oil refineries, and soccer fields from geospatial images.Other image analysis software modules may have identificationrepertoires that differ partially or completely from the first imageanalysis software module as the invention Physically, the image analysissoftware module can be present on the same computer as the deep learningmodel training module 520, or could be placed anywhere in the world, asonce training is complete, no connection between the server running thedeep learning module training module 520 and the image analysis softwaremodule 540 is necessary. The trained convolutional neural network caneven be programmatically cloned such that a plurality of instances ofimage analysis software modules 540 with the same object identificationrepertoire may be present concurrently with no specific geographicalrelationship, the only stipulation being that there is access to a cacheof orthorectified, multi-scale, geospatial image segments tagged withinformation that allows the geographical location of image capture andthe scaling factor to be determined 530. In a preferred embodiment ofthe system, image correction filters 541 taken from a set that compriseshistogram normalization, edge enhancement and cloud detection. A clouddetection filters is present to allow removal of images when occlusionprevents analysis of the ground below. These filters 541 are depicted asapplied on the same computer on which the image analysis software moduleresides but filters can be applied on any remote computer that islogically between the cache of multi-scale unanalyzed geospatial imagesegments 530 and the image analysis software module 540. During imageanalysis, geospatial image segments of the same region but differingscale factors are scanned by sliding window multi-scale analysis module542 which converts between any internal coordinate system used by thecache of geospatial images in use to geographical latitude and longitudewhile tracking all changes in fine positioning coordinates as the windowof focus changes. While each image segment is scanned, all features ofinterest that are identified are given a confidence factor numberreflecting correspondence between the newly detected item and the imageanalysis software module's set of characteristics for the item. Featureitems of interest detected by the image analysis software module 540during the scan of geospatial image segments from cache 530 are thenreported by the system 550 in a format specified by the requester. Asexamples, reports may be in the form of appropriately scaled geospatialimages tagged with either tokens or color shading which represent eachfeature item of interest, or may be rows of text providing suchinformation as item type, latitude and longitude, but may take anyformat dictated by the needs of the analysis and known to the art.

Description of Method Embodiments

FIG. 6 is a process flow diagram of a method 600 for geospatial imageanalysis that uses the deep learning model and a convolutional neuralnetwork using a system of the invention. According to the embodiment,orthorectified geospatial image segments that have been previouslyanalyzed and where feature items of a specific type which may include,but are not limited to trucks, fighter aircraft, oil wells, missilelaunchers or elephants have been clearly demarcated and labeled areretrieved from a cache of images 601. The mechanism that results in thelabeling of specific objects within geospatial image segments availablefrom the cache 601 whether that mechanism be analysis and labeling bytrained image analysts, intended for use in the method 600 or for someother purpose but then re-purposed for the method 600; those segmentsare labeled by a crowdsourcing where a plurality participants, paid orvolunteer, analyze the same image segments from a set of image segmentsin campaign to identify and precisely demarcate the same feature itemand then image segments where the labeling accuracy is calculated to bestatistically very high are placed in the cache 601; or labeled by somemeans not listed; should not be seen to limit method 600 as the methodis envisioned to utilize all of those mechanisms either separately ortogether. As feature item types that might be of interest to therequester of geospatial image analysis often can be further divided intomultiple subtypes, the feature items are frequently, though not always,placed into categories 602 prior to training of image analysis softwaremodule 540. For example, a “truck” could be “dump truck”, “deliverytruck”, “tanker truck” “tractor tailer truck” or “tow truck”, to namejust a few. All of these “trucks” appear different in geospatial imagesand for the method 600, “trucks” of the same subtype (“delivery truck”)but with widely different paint schemes might vary sufficiently torequire additional cues be added prior to system training 609. Themethod 600, therefore, accounts for the need that geospatial imagesegments may need to be categorized by the generalized form of objectthat is labeled 602. One with ordinary skill in the art will understandthat all categories used by method 600 are arbitrary and the discretelevel of categorization depends upon the specifications of the search.For instance, using the “truck” example, one could envision anembodiment of the method 600 where the requestor may want to identifyonly “tanker trucks”, so there would be no “truck” category. In anotherembodiment of the method 600 the requestor may want to identify “trucks”in general and “dump trucks” as a separate group. Whether placed intocategories 602 or not placed into categories, orthorectified geospatialimage segments that have been labeled to identify feature items whichwill be used to train the image analysis software module 540 may beoptimized for the training process through the application of digitalimage filters 603. The scale or resolution of images to be used intraining will be determined from metadata included with the imagesegments 604. Adjustment of image segment scale 603 might then occurbecause training the image analysis software module 540 relies on imagesegments bearing instances of the target feature item of interestpresenting that object within narrowly controlled size range. Inembodiment of the method 600, color correction filters may also beapplied to image segments to be used to train the image analysissoftware module 540. One modification commonly applied to images used indeep learning model training 607 of convolutional neural networks usedin image analysis software module 540 is conversion of color images tograyscale images 605 which serves to simplify the image significantlyand also greatly simplifies the application of subsequent filters thatmay be used to optimize image for training. Color images may beconverted to grayscale by first converting the image from the RGBcolorspace to the YCrCb colorspace and then discarding all but Y channeldata which results in a grayscale image of tonal quality known to workwell in deep learning model training of convolutional neural net used inthe invention. Other methods of color to grayscale conversion that mightbe used are linear desaturation by averaging the R, G, and B intensitiesof each pixel into a single grayscale intensity value, and weightedconversion where the intensities of the R, G, and B are multiplied bydifferent weighting values based upon a pre-determined desired endresult and those weighted values then averaged into on single grayscaleintensity value. Methods of color to grayscale image conversion 605outlined here are meant only to be exemplary and should not beinterpreted to limit conversion methods that could be used, includingthe absence of color to grayscale conversion in the training imagepreparation process. Another type of image correction 605 that may beemployed to prepare pre-labeled geospatial images for use in deeplearning model training 607 is histogram normalization which oftenincreases image contrast and serves to reduce the effects of differencesin lighting during exposure by producing sets of image segments withsimilar dynamic range profiles prior to use in training of the imageanalysis module 540. Examples of histogram normalization filters thatmight be used to prepare geospatial images for training are linearhistogram normalization, histogram equalization, adaptive histogramequalization, and contrast limiting adaptive histogram equalization. Oneskilled in the art will realize that while the use of these imagehistogram manipulation methods may produce image segments significantlybetter suited for the supervised stage of deep learning 607, the method600 described herein does not absolutely rely on histogram normalizationin its generalized form or of the exemplary histogram manipulationmethods specifically. In addition to, or in lieu of, those methodsmentioned in some depth, image filters such as Gaussian, median filter,and bilateral filter may be applied to enhance edge contrast ofpre-labeled geospatial image segments as is common in the art, howeverthe listing of these filters is meant only to provide further examplesand should not be taken to bar the use of other filters that are notlisted as part of the invention or to suggest that the method 600 relieson any of these filters.

As is appreciated by those skilled in the art, much of the recentsuccess in the recognition of specific objects in digital images bycomputers using trained convolutional neural networks is due to the useof the deep learning method 607. Under this method of training, theconvolutional neural network is first trained with a set of images wherethe item to be identified, ideally in collocation with a large pluralityof backgrounds and in a large plurality of orientations, is clearlydemarcated digitally. This stage of deep learning is referred to by theart as “supervised.” The number of images where the object of interesthas been clearly demarcated is often limited and often gives rise totrained convolutional neural networks that are excellent in recognizingthe object of interest in the training set but otherwise recognize theobject of interest poorly. Object recognition reliability issignificantly increased by exposure of the convolutional neural networkto sets of images 606, much larger in image number than thelabeled-training set, where the object of interest is present but notlabeled or where the object may not be present. This stage of learningis referred to by the art as “unsupervised.” One approach to generatelarge sets of images 606 for unsupervised learning is to screen verylarge repositories of general images for suitable examples. Anotherapproach of generating the unsupervised learning image set 606 is tostart with the set of images used for supervised learning stage 601 andmodify them by linearly translating or rotationally transforming theobject of interest; changing the appearance of the object of interest orbackground through image histogram modification, Gaussian filter,bilateral filter or median blur filter 603; or both using objectplacement and filter modification to create a plurality of images 606useful in the unsupervised learning stage from images 601 used in thesupervised learning stage. The method 600 depicted here can usegeospatial images generated by either approach, identification of newimages, or manipulation of the supervised learning image set, even somemixture of both methods, in the deep learning model training 607 of theautomated image analysis module 540.

Once trained, image analysis software module 540 can and often does havethe ability to identify multiple categories of objects of interest 613from caches of unanalyzed geospatial image segments 608. A first set maycomprise buildings, tents, cars, airliners, oil refineries, and soccerfields. Another set may comprise zebra, elephants, tents, villages,light aircraft, and trucks. Items listed are merely a very short list ofeasily imagined examples, the model described herein allows theidentification of any object that can be discerned from geospatialimages by all means known to the art either with or without the use ofdigital image filters 609 which may be selected from a set comprisingcolor to grayscale conversion, histogram correction or equalization, orcontrast enhancement filters 610. Image segments may also be passedthrough cloud detection filter 611 which analyzes images for cloud coverthat occludes a percentage of the image deemed to make the imageunsuitable for further analysis and removes those images analysis 612.Model 600 can also allow for the automated identification and locationdetermination of feature items of interest that vary greatly in size byusing caches of unanalyzed source images that contain image segments ofthe same region stored at multiple scales. Sliding window software 614that is part of 600 scans the images that are at different scales andprecisely keeps track of the geographical coordinates by convertingscaling factor and internal segment coordinate system information of theimage segment cache into latitude and longitude. This, for instance, forthe purpose of example only, a single automated image analysis module540 might locate all of the cities and oil wells within the same regionduring a single analysis. By using caches of image segments taken of thesame region over known intervals of time, the system might also be usedto track migrations of animal herds, the movement of groups of people,or the movement of equipment, to name just a few of the many possibleexamples that one skilled in the art might envision, which are ofsignificant interest but for which occupying the scarce and potentiallycostly talents of image analysts makes the study impractical. In alltypes of analysis, the system will generate some form of report 615. Theformat of the report, whether it be a set of geospatial images marked insome manner to show the locations of items of interest, numeric outputshowing some code for each item category plus latitude and longitudecoordinates, item codes with internal geospatial image segment cachecoordinates which might also include segments rejected due to imageocclusion or any other format either physical or electronic which isknown to the art and desired by the requestor of the survey as thesystem does not depend on one particular form of output.

FIG. 7 is a three-panel example 700 of cloud detection by imagecorrection and optimization module 541 depicted in the exemplary systemarchitecture block diagram of FIG. 5. When system 500 loads an image 701from the cache of multi-scale unanalyzed geospatial image segments 530for analysis, several image filters from a set including, but notlimited to, color correction, artifact removal and resolution datadetermination may be applied. One such filter is specifically written todetermine whether one or more portions of the field of view of theground are occluded, for example by thick cloud cover. This softwarefilter identifies subregions of the current working geospatial regionthat are obstructed by cloud cover using characteristics common toclouds in such images which include but are not limited to brightness,color uniformity and lack of image detail. After establishing the edgesof cloud using one of the many edge detection methods known within theart, the filter changes all pixels constituent to the cloud to a singlecolor value 702 to clearly demarcate the portion of the image that isobstructed. Further software within the image correction andoptimization module 541 may then reject the image from image analysisfor object identification 540 based upon a pre-selected threshold 703.

FIG. 8 is a three-panel diagram 800 depicting the operation of exemplarysliding window search and locate subroutines that are part of imageanalysis software module 540 of the invention depicted in FIG. 5.Establishment of the location of any identified objects of interestgenerally requires that conversion of the coordinate system usedinternal to the cache of multi-scale unanalyzed geospatial imagesegments 530 to the coordinates of earth latitude and longitude takesplace. Cached orthorectified geospatial image segments, or tiles 810,are regularly stored in caches using cache-internal coordinate systems811, 812 of which the tile map service system and the web map tileservice are two standardized examples that the invention mightencounter. It should be clear, however, that the mention of these twosystems should not be construed as the only possible coordinate systemsthat method 600 might use, as the invention does not depend on anyspecific system and tiles encoded by any coordinate system known tothose skilled in the art may be used, provided that that system suppliesa conversion pathway to geographical latitude and longitude and alsoincludes scaling information. In one embodiment of this design, theimage cache service might provide internal row and column numbers of animage segment's origin 813 in addition to a scaling factor of thesegment that is supplied. From this segment-specific information and anyadditional cache coordinate system specification information, slidingwindow subroutines of the invention can easily and accurately convertfrom the cache's coordinate system to standard geographical longitude821 and longitude 822 for any point of the an image segment that isundergoing analysis 820. Specifically this is done by subroutines in thesliding window software module which, as the window of focus scans afirst image segment for objects of interest 830, keeps track of thechange in latitude and longitude using the equation φ_(new)=a sin(sinφ_(old)×cos δ+cos φ_(old)×sin δ×cos θ) for change in latitude and theequation λ_(new)=λ_(old)+a tan 2(sin θ×sin δ×cos φ_(old)×cos θ−sinφ_(old)×sin φ_(new)) for longitude where φ is latitude, λ is longitude,θ is bearing (clockwise from north) and δ is angular distance traveled((scale corrected distance traveled/radius of the earth)) 831.

FIG. 9 is a two-panel FIG. 900 showing two examples of the results thatmight be obtained upon analysis of image segments from one or morecaches of orthorectified image segments by exemplary trained imageanalysis software module 540 of the invention. The left panel 910 of 900shows an orthorectified geospatial image that has been scaled and taggedwith yellow to allow those who receive the report to confirm a groupingof airliner category objects 912, 913, 914, 915. The legend for thispanel 911 shows that image analysis software module in one embodiment ofthe system was trained to identify a plurality of object categories someof which were airliner, building, fighter aircraft, helicopter, road andrefinery tank. This listing is exemplary of what might be part of aparticular search report of this type and does not indicate that thoseitems encompass the entire identification repertoire of the trainedimage analysis software module 540 employed to generate the report asone skilled in the art will understand that the listed categories areonly a small subset of possible categories that might be identified bythe invention. The right panel of FIG. 9, 920, again shows anorthorectified geospatial image that has been scaled and tagged thistime with purple to allow those who receive the report to confirm agrouping of fighter aircraft category objects 922, 923, 924. The legendfor this panel 921 shows that image analysis software module in thisembodiment of the system was trained to identify a plurality of objectcategories that include airliner, bridge, fighter aircraft, helicopter,refinery tank, soccer field, baseball field. As in Panel 910, thislisting is meant only to be exemplary of what might be included in aparticular type of search report and not to indicate that those itemsencompass the entire identification repertoire of trained image analysissoftware module 540 used in the system. That the format of the reportsfound in FIG. 9, an orthorectified geospatial image scaled to showindividual airliners or fighter aircraft and color coded to highlightthose items as objects of interest, is just one easily discerned exampleof report types that could be generated by the system and is meant assuch. These reports can take a plurality of forms depending uponrequirements of the specifications of those who perform the search. Onesearch might produce a report made up solely of rows of numbers, one todesignate the specific object of interest that has been identified, asecond for and third the latitude and the longitude at which that objector objects were found. Another embodiment might use further logic toproduce identify the probable function of a particular grouping ofobjects. A grouping of airliners within a given are might get a singletag as an airport, or a grouping of automobiles tagged as a parking lot.When the invention is used to analyze the same specific region over someperiod of time, a report text only, tagged image of combinations of bothmay be generated only to record changes in the numbers of specific itemsdrawn from a vast plurality of categories such as tents, cars,submarines, zebra, or tree cover. The invention is not constrained ordefined by any particular report type.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for broad area geospatial objectdetection using auto-generated deep learning models, comprising: a deeplearning model training software module stored in a memory of andoperating on a processor of a computing device; and an image analysissoftware module stored in the memory of and operating on the processorof the computing device; wherein the deep learning model trainingsoftware module: receives training data comprising a plurality oforthorectified geospatial images with a plurality of objects presenttherein, at least a first subset of the plurality of objects beinglabeled and a second subset of objects being unlabeled; classifies thetraining data into a plurality of categories; applies one or more imagemodification steps to the training data; and generates an objectclassification model from the training data using a deep learning methodcomprising separate processing of the first and second subsets of thetraining data through a convolutional neural network system; and whereinthe image analysis software module: receives orthorectified geospatialimagery; applies a plurality of image modifications to the unanalyzedorthorectified geospatial imagery; discards images unsuitable foranalysis; uses the object classification model to automatically identifyand label all objects of interest in the received, unanalyzedorthorectified geospatial imagery; and outputs the geospatial locationsof the identified objects of interest.
 2. A method for conductingautomated identification of feature items from orthorectified geospatialimagery data comprising the following steps: receiving, at a deeplearning model training software module stored in a memory of andoperating on a processor of a computing device, a plurality oforthorectified geospatial images, divided into discrete segments where aplurality of objects of interest are present therein, at least a firstsubset of the plurality of objects of interest being labeled and asecond subset of objects of interest being unlabeled; placingpre-labeled geospatial imagery segments into categories based upon theobjects of interest they contain; applying one or more imagemodification steps; generating, using a convolutional neural networksystem operating as part of an image analysis software module stored ina memory of and operating on a processor of a computing device, anobject classification model from the training data using a deep learningmethod comprising separate processing of the first and second subsets ofthe training data; using the trained image analysis software module toidentify feature items of interest within unanalyzed geospatial imagerydata; and reporting the identities and geospatial locations of aplurality of features of interest.
 3. The system of claim 1, wherein theplurality of categories are pre-defined.
 4. The system of claim 1,wherein the plurality of image modification steps are drawn from a setcomprising artifact removal, color standardization, and resolution datadetermination.
 5. The system of claim 1, wherein the objects of interestare automatically identified and labeled regardless of the orientationof that feature item within the section.
 6. The system of claim 5,further wherein the objects of interest are automatically identified andlabeled accounting for differences in item scale by using a multi-scalesliding window algorithm.
 7. The method of claim 2, wherein theplurality of categories are pre-defined.
 8. The method of claim 2,wherein the plurality of image modification steps are drawn from a setcomprising artifact removal, color standardization, and resolution datadetermination.
 9. The method of claim 2, wherein the feature items ofinterest are identified regardless of the orientation of that featureitem within the section.
 10. The method of claim 9, further wherein thefeature items of interest are identified accounting for differences initem scale by using a multi-scale sliding window algorithm.